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The competition between Kitaev and Heisenberg interactions away from half filling is studied 
for the hole-doped Kitaev-Heisenberg t-Jx-Jn model on a honeycomb lattice. While the isotropic 
Heisenberg coupling supports a time-reversal violating d-wave singlet state, we find that the Ki- 
taev interaction favors a time-reversal invariant p-wave superconducting phase, which obeys the 
rotational symmetries of the microscopic model, and is robust for Jh < Jk/2- Within the p-wave 
superconducting phase, a critical chemical potential = fi c « t separates a topologically trivial 
phase for < fi c from a topologically non-trivial Z2 time-reversal invariant spin-triplet phase for 
I A* I > Mc- 
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The concept of topological order, originally introduced 
in the context of quantum Hall systems, has recently been 
extended to topological band insulators and topological 
superconductors [TJ [2]. One of the exciting properties 
of these systems is the prospect of observing non-abelian 
quasi-particles. A prototype model for non-abelian quasi- 
particles is the Kitaev model on the honeycomb lattice 
with a non-abelian topological phase [3]. 

There is both experimental [U [5] and theoretical [6] 
evidence that a Kitaev model with an admixture of anti- 
ferromagnetic Heisenberg exchange is realized in iridates 
(Li2ir03 and Na2lrOa) with a honeycomb lattice. For 
dominant Kitaev coupling, the ground state is a spin 
liquid. Upon increasing the relative strength of the 
nearest-neighbor Heisenberg coupling, the ground state 
first turns into a stripy antiferromagnetic phase and then 
into a Neel antiferromagnet [7J [5]. Both Li2lrC>3 and 
Na2lr03 are found to be magnetically ordered at low tem- 
peratures [U IMTT] . The temperature dependence of 
the magnetic susceptibility [H H] and the recent resonant 
x-ray and neutron scattering experiments [SHU] indicate 
that the ground state of Na2lrC>3 is most likely described 
by a zig-zag spin structure. This type of magnetic order 
can be theoretically explained by including further neigh- 
bor Heisenberg couplings [5j [12] or in terms of trigonal 
distortion of the oxygen octahedra [T3]- On the other 
hand, based on the magnetic susceptibility measurements 
for Li 2 Ir03, it has been suggested that Li 2 Ir03 may be 
close to the Kitaev spin liquid regime [5]. 

The Kitaev spin model can be viewed as a Mott in- 
sulator at half filling, and one can ask about its phase 
diagram away from half filling. It is known that a 
Heisenberg antiferromagnet on the honeycomb lattice be- 
comes a d-wave superconductor upon doping [T3] . Hence 
one may expect superconductivity in the doped Kitaev 
model, and can ask how the superconducting order pa- 
rameter depends on the relative strengths of antiferro- 
magnetic Heisenberg exchange and the Kitaev interac- 



tion. In this paper, we address this question by studying 
the Heisenberg-Kitaev model in a slave boson theory, and 
find that the Kitaev model turns into a p-wave supercon- 
ductor upon doping, with an internal spin-orbital struc- 
ture of Cooper pairs as illustrated in Fig.[]Ja). When the 
doping level is larger than one quarter, this superconduc- 
tor is an example of a time-reversal invariant Z2 topo- 
logical superconductor [T5H22] . which supports a pair of 
counter-propagating helical Majorana modes at the edges 
of the sample and one pair of Majorana zero modes at 
the vortices. This p-wave superconductor is considerably 
more robust to adding a Heisenberg exchange than the 
spin liquid phase itself at zero doping, thus raising the 
possibility that it might be observable in doped iridates. 
We also find that the time-reversal invariant topologi- 
cal superconductivity is not sensitive to the details of 
the band structure. Thus, the physics discussed here 
should be representative for a broad class of materials 
with strong spin-orbit coupling. 

We consider a honeycomb lattice with three incquiva- 
lcnt nearest-neighbor bonds referred to as 7 = x, y, z [see 
Fig. [jjb)]. The Kitaev-Heisenberg model with nearest 
neighbor hopping is given by 

H = -J K J2s;'S] + J H J2(srS 3 -^f-)+H T . (1) 

The first term in the Hamiltonian ([lj is called the Ki- 
taev interaction [3] , and it describes an Ising-like coupling 
between the 7 components of spins = \f\ oS^apfifi 
at each bond in the 7-direction. The second term de- 
scribes an isotropic Heisenberg interaction with interac- 
tion strength Jh, and Ht is the kinetic term 

H T = -t ]r ./;;„./• • h.c. . (2) 

<ij>,a 

We assume that at half-filling the system can be viewed 
as a Mott insulator and adopt the so-called U(l) slave 
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FIG. 1: (color online) (a) Relative orientations of the Cooper 
pair orbital (thick arrows) and spin (thin red arrows) angular 
momentum in the ground state wave function for topologi- 
cally nontrivial time-reversal invariant p-wave pairing. Con- 
ventions for (b) real space and (c) reciprocal lattices. The 
four time reversal invariant points in the Brillouin zone are 
r, Mi, M 2 , and M 3 . 



boson method to take into account the renormalization 
of the hopping amplitude by strong correlations [23] [24] • 
By assuming that the holons are condensed to give a co- 
herent Fermi-liquid state, the hopping amplitude t can 
be written as t = toS, where to is the bare matrix ele- 
ment and 8 quantifies the hole doping so that 1 — 8 is the 
average number of electrons per site. 

We introduce a spin-singlet and three spin-triplet oper- 
ators, defined respectively as = (fa fa - fafa)/V%, 
kj,x = (fa fa- fa fa)/ V^, Uj, y = i(fafa + fafa)/v2> 
Uj.z = i(fafa + fafa)/V%, and re-write the Kitaev- 
Heisenberg part of the Hamiltonian in terms of these op- 
erators. The Kitaev term on an x-link is given by 
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b ij,x L n,x L ij,y°t],y L ij,z L i3,z\ 



(3) 



and for y- and z-links the plus sign appears in front of 



the t]j y tij^y and tj z tij lZ operators, respectively. 

The mean field Hamiltonian is obtained by replacing 
the spin-singlet and spin-triplet operators by their expec- 
tation values in the usual fashion. We denote the spin- 
singlet and spin-triplet expectation values by A 7 and 
respectively, where 7 corresponds to the bond direction, 
and p the component of the mean-field triplet vector, 
such that 



A. 



dP 
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for 7 bonds, 



4V^ 



(h 



for 7 bonds. 



(4) 



A different decoupling of the Kitaev interaction into both 
hopping and pairing channels was used in [25j to describe 
the undoped Kitaev model. Here we consider only the 
pairing channel, which is justified in the limit of reason- 
ably large doping. 
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FIG. 2: (color online) The transition temperature ksTc/to 
in the p-wave triplet channel as a function of the Kitaev 
interaction strength Jn/io for different doping levels 8 = 
0.1,0.25,0.5, at J H = |3D]. 



At the critical temperature for the superconducting 
phase transition the order parameter vanishes and we ob- 
tain linearized gap equations [25]. The spin singlet case 
was previously solved in Ref. 1141 obtaining a mixed su- 
perconducting phase of d-wave intraband pairing and p- 
wave interband pairing. Following a similar procedure we 
obtain linearized gap equations also for the spin-triplet 
order parameters. We find that the d? with different p 
are not coupled in these equations and therefore we can 
conveniently write the linearized gap equations for triplet 
order parameters using three stability matrices M x>VjZ . 
The critical temperatures for the different channels can 
be obtained by finding the largest temperature where at 
least one of the eigenvalues of the stability matrix is equal 
to 1. The possible solutions of the order parameter near 
the critical temperature are linear combinations of the 
eigenvectors corresponding to this eigenvalue. 

We find that the stability matrix for is 



M-r. 




(5) 



The stability matrices M y and M z can be obtained from 
M x by cyclically changing the column of negative signs. 
The matrix elements are B — Ai—j and C — A^j , where 



.4; 



2N 



/tanlK lanhl U ;2 /2i 



26 



26 



• sin((5 i • q — cf>) sm(Sj ■ q- 

sinh(/3 c /i) cos(8i ■ q — 4>) cos(8j ■ q- 
2/xcosh(&6/2) cosh(/3 c £ 2 /2) 



(6) 



in which /3 C = l/(fcsT c ), Si denote the nearest neighbor 
vectors, £1(2) = 4=|i(<7*)| — M are the single particle disper- 
sions, t(q) = t^2j e lSj 'Q and <fi = a,ig[t(q)}. 

Because of the symmetry relation of the matri- 
ces M XtViZ , the eigenvectors of the linearized gap 



3 




1.2 



°' 5 Jn/to 



FIG. 3: (color online) Phase diagram for the condensation in 
the spin-singlet (red/dark gray regions marked as s-SC and 
d-SC) and spin-triplet (green/light gray regions) channels as 
a function of Heisenberg interaction Jh and doping S, for Ki- 
taev interaction strength Jk = to. In the singlet channel 
both d-wave and s-wave order parameters can exist depend- 
ing on the doping and interaction strength. The d-wave order 
parameter breaks the time-reversal symmetry. Within the p- 
wave triplet phase, there is a topological phase transition (in- 
dicated by dashed line) between topologically trivial 8 < 0.25 
and topologically non-trivial S > 0.25 time-reversal invariant 
superconducting states. The (blue/dark gray) region in the 
upper left corner of the phase-diagram denotes the parameter 
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FIG. 4: (color online) The minimum energy required to ex- 
cite quasiparticles as a function of the dimensionless inter- 
band order parameter strength e and chemical potential 
for 77 = 0.042£. The interband order parameter is adiabati- 
cally switched on when e changes from e = to e = 1. The 
inset demonstrates that for each value of e the energy gap 
closes in the vicinity of = t indicating a transition be- 
tween the topologically trivial and nontrivial states. 



regime where k B T c < 10" 4 t - 



equation with largest critical temperature are three- 
fold degenerate. By denoting the solutions as d = 
{d%, dy , d% , dtj., d|, df , d% , d* , d*) , the linearly independent 
solutions are 

di = (0,-1,1,0,0,0,0,0,0), 
d 2 = (0,0,0,1,0,-1,0,0,0), 
d 3 = (0,0, 0,0, 0,0, -1,1,0). (7) 

We now compare the critical temperatures for super- 
conducting condensation in the spin-singlet and spin- 
triplet channels. In Fig. [2] we show the critical temper- 
atures for the onset of spin-triplet superconductivity at 
various doping levels. In the spin-singlet channel, the 
Kitaev interaction affects the pairing only by renormal- 
izing the interaction strength Jh and therefore the criti- 
cal temperatures for singlet-superconductivity can be ob- 
tained using the theory of Ref. Q3] that predicts a time- 
reversal violating d-wave state (analogous to that found 
for a doped Heisenberg model in a triangular lattice [27l - 
[29] ) or s-wave state depending on the interaction strength 
and doping. The phase diagram is shown in Fig. [3j 
where the dominant condensation channel is computed 
as a function of doping and Heisenberg interaction Jh 
for fixed Kitaev interaction strength Jk = to. Interest- 
ingly, we find that the phase transition between triplet 
and singlet superconductivity appears at the relative in- 
teraction strength Jh / Jk — 1 /2. We note that in the ab- 
sence of doping the topologically interesting Kitaev spin 



liquid phase exists only in the range Jh < Jk/8 [H]. 
This indicates that the spin-triplet superconductivity is 
very robust to the presence of the Heisenberg interaction, 
and therefore might be observable in doped iridates. 

The order parameter for triplet superconductivity at 
temperatures below the T c can now be calculated from 
the nonlinear gap equations. We assume that the order 
parameter is a linear combination of the three degenerate 
solutions to the linearized gap equations, and so we can 
write d = (ryidi + r)2&2 + %d3). Solving the gap equa- 
tions iteratively from different initial conditions, we find 
four solutions (771 , 772, 773) = r/(l,±l,±i). These solutions 
have large basins of attraction. Moreover, the real parts 
of the eigenvalues of the stability matrices obtained by 
linearizing the gap equation around these solutions are 
smaller than one, indicating stability. Within our mean 
field approach, the magnitude of 77 depends on tempera- 
ture, doping, and the interaction strength Jk [30] ■ For 
k B T = \Q- 4 t, I /j I = 1.2t (corresponding to 6 « 0.38) and 
Jk = 2.3to, we obtain 77 w 0.042i. This value of 77 is used 
for Figs, [i] and [5] 

By expressing the intra- and interband order param- 
eters using the convention i(d(<f) • a)<72 [26], we obtain 
expressions for the intra- and interband d-vectors in mo- 
mentum space. Expanding these solutions around the T 
point, we obtain 
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FIG. 5: (color online) Integrated density of states for r\ = 
0.042t and chemical potentials 0.7t, fj, c , and 1.3t. The critical 
value of the chemical potential is /i c = 0.9932f. 



di2® = ^ + 2V3q x q y + 3q 2 y , (8) 

± (-3^ - 2\F&q x q y + 3q y ), =F4%/3g x q 1 , J , 

where d 22 = — dn are the intraband pairing vectors, and 
d 2 i = di2 are the interband pairing vectors. Using 
these dij(g)-vectors we find that the numerical solution 
of the nonlinear gap equations is that of a time-reversal 
symmetric odd parity phase in the dominant intraband 
channel, and a time-reversal symmetric even parity phase 
in the interband channel. Both order parameters retain 
the six-fold rotational symmetry of the lattice, and so 
the phases obtained obey all the underlying symmetries 
of the microscopic model [31]. 

The order parameter dn in Eq. ^ lies on a circle with 
fixed |dn| when q rotates around a circle in the (q xi q y )- 
plane. The axis around which dn rotates depends on 
the choice of signs in Eq. For example, the choice 
(— ,+) corresponds to a rotation axis along the (1, 1, 1) 
direction. By virtue of a global transformation of the spin 
quantization axis in the model Eq. ([!]), the dn vector 
is rotated into the xy-plane, and describes a q x — iq y 
pairing for spin-up and a q x + iq y pairing for spin-down 
(equivalent to the B-phase of superfluid 3 He) components 
of the order parameter. In the original model Eq. 0, this 
corresponds to q x — iq y pairing for spins pointing in the 
(1, 1, 1) direction and q x + iq y pairing for spins pointing 
in the (—1, —1, —1) direction, see Fig.[l|a). 

In the absence of interband coupling and in the weak 
pairing limit, the value of the i? 2 invariant of a time- 
reversal invariant topological superconductor is deter- 
mined by the topology of the Fermi surface 18 - 22 . More 
precisely, its value is determined by the parity of the 
number of time-reversal invariant points below the Fermi 



level. On the honeycomb lattice, the Fermi surface topol- 
ogy changes from hole pockets around the two inequiva- 
lent K points to a particle-like Fermi surface around the 
r point when the chemical potential is tuned through the 
critical value \fi\ = \i c = t (corresponding to the critical 
value of hole doping S c — 0.25). For < fi c , there are 
four time-reversal invariant points below the Fermi level 
(the three M points and the T point shown in Fig. [I]), 
and the superconductor is in a topologically trivial phase. 
At the special point |/x| = fi c in the single particle dis- 
persion, the Fermi surface touches the three inequivalent 
time-reversal symmetric M points. As a consequence, for 
|ju| > /i c , the only time-reversal invariant point enclosed 
by the Fermi surface is the T point. Because the intra- 
band order parameter has odd parity, the superconduc- 
tor in the absence of interband order parameter is guar- 
anteed to be a topologically non-trivial superconductor 
[I8ll22j . in the class Dili [15] • Adiabatically switching 
on the interband pairing, we find that the quasiparti- 
cle dispersion indeed remains gapped in each topolog- 
ical regime and that the value of the critical chemical 
potential /x c , where the quasiparticle dispersion becomes 
gapless, is renormalized downwards. Therefore, the topo- 
logical phase transition remains robust in the presence of 
the even parity interband phase (see Fig. [4]). This phase 
transition is shown with a dashed line in Fig. [3j 

The topological phase transition can be observed by 
measuring the density of states as a function of energy, 
e.g., by NMR and tunneling experiments. Fig. [5] shows 
the density of states as a function of energy for three 
different values of chemical potential. Above and below 
| /i | = fx c the density of states vanishes at low energies 
resulting in an energy gap, which is of the order of r/. On 
the other hand, at the phase transition \fi\ = fi c the en- 
ergy gap closes resulting in an increased density of states 
at low energies. 

A few remarks are in order concerning the relevance 
of our results for the actual materials. First, we have 
considered here the nearest-neighbor hopping Hamilto- 
nian, but our results are in fact robust with respect to 
changes in the band structure. We have verified that 
all our qualitative results remain valid if a second near- 
est neighbor hopping t' = O.lt is introduced. Second, 
the necessary values of the Kitaev interaction strength 
for the existence of topological p-wave superconductivity 
might seem quite large Jk > 0.7t . Nevertheless, these 
values are not unrealistic for the iridates, because the 
relation between the exchange interaction strength and 
the effective nearest-neighbor hopping in iridates is not 
as simple as, for example, in cuprates. In systems with 
strong spin-orbit coupling, the electron collects a phase 
factor during the superexchange process giving rise to a 
Kitaev interaction, and, at the same time, the effective 
nearest-neighbor hopping amplitude is strongly reduced 
by negative quantum interference. We also notice that 
approximately similar values of the exchange interaction 
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strength have been discussed for graphene [13] . 

In summary, we have studied the competition of spin- 
singlet d-wave superconductivity and spin-triplet p-wave 
superconductivity in the hole-doped Kitaev-Heisenberg 
model on a honeycomb lattice. We have shown that 
the Kitaev interaction favors a time-reversal invariant 
topological superconductivity, which is robust to the 
presence of an isotropic Heisenberg interaction Jjj < 
Jk/%- Because this topological phase supporting pairs 
of counter-propagating Majorana modes persists over a 
much wider region of the phase diagram than the Kitaev 
spin liquid phase itself in the absence of doping, it might 
be easier to find materials where it becomes observable. 

We would like to thank A. Vishwanath and A. Schny- 
der for useful communications and comments. Financial 
support by the BMBF is acknowledged. 
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